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In the beginning: Control by direct pursuit or compensatory tracking 


The user interacts: The real world affords 


Objects are manipulated (=> Actions, interventions, modifications 
according the the user's Feeds back: resistence, resilience 
intentions 


monitoring 


The first phase: pursuit and compensatory tracking via indicators and actuators 


Technology as a tool: 
AII variables are represented by 


indicators and manipulated by 
actuators 


informing 
manipulating 
Feed-back 


The user interacts: 
Objects are manipulated 
according the the user's 
intentions 


The real world affords 
Actions, interventions, modifications 
Feeds back: resistence, resilience 


Yellow arrows indicate actions not perceivable by the user 


The number of instruments in the 
cockpit over time 


>3000 


1904 Concorde 


The Paradox of the Artificial Horizon 


* In flight, when using the actuators you 
intend to influence the behavior of the plane 


however 


е the instrument (artificial horizon) makes you 
believe that you influence the world outside 
while the plane remains static 


The second phase: pursuit and compensatory tracking via displayed representations 


Technology as a tool: la Technology as a representation 
AII variables are represented by \ _ - | of situations: 

indicators and manipulated by all relevant situations are integrated 
actuators in displays 
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The user interacts: The real world affords 
Objects are manipulated Actions, interventions, modifications 
according the the user's Feeds back: resistence, resilience 


intentions 


Yellow arrows indicate actions not perceivable by the user 
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The possible future: direct manipulation of virtual objects in a virtual space 
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Yellow arrows indicate actions not perceivable by the user 


Pilot's virtual view 
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Consequences for learning 


e What you see is what you can act upon 


* The actions on your fingertips result in the 
appropriate actions on the selected objects 

* The behavior of the virtual objects in the virtual 
space correspond to your mental model of the 
environment 

* The effects of your actions are directly fed back by 
the visual, acoustic and/or tactile channel, that is 


learning becomes direct instead of mediated 


THANK YOU FOR YOUR ATTENTION 


Now, | would appreciate questions or 
comments 


NTNU - Trondheim & 
Norwegian University of О 
Science and Technology 


New Trends in Social Robots 
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* What is social robot? 

* Possible applications of social robot? 
* Challenges 

* User profile 

e Robot personality 

* Conclusions & discussions 


What is social robot? o 6 


e A social robot is an Al (artificial intelligence) system, such 
as Android, that is designed to interact with humans and 
possibly also with other robots. 

e Social and personal robots are a type of mobile robots 
that look like, communicate and interact like humans and 
are used where people used to live. 

e It should be able to verbally and non-verbally (facial 


expressions, posture, nodding, eye-contact, gesture, 
waving, etc.) communicate with humans. 


Possible applications of social robot? o @ 


They are used to replace - in part — visiting human care 
givers in elderly/nursing houses in their roles of distributing 
and ensuring medication is taken and reporting back if they do 
not see the medicine being swallowed 


As a school teacher for languages and other subjects 


For replacing real assist “seeing” and “hearing” animals for 
blind people to quickly clarify the situation and provide live 
feed 


For childcare at homes and kindergartens 
To help people with reduced mobility 

As office receptionist 

Replace real animals with robotic animals 
etc. 


Market value of personal and service © € 
robots _ 


e By 2030, many households in the developed world will 
have personal robots in their home. 

* Personal robots tomorrow will be like personal 
computers today. 

* According to a Forest & Sullivan’s recent study, the 
future of mobile robots, the market for mobile robots is 
expected to reach $17.4 billion by 2020. 


Challenges o ( 


e The challenge is to have robots that can understand 
human emotions, expectations and needs and react 
accordingly. 

e Integration of artificial intelligence, deep learning, 
cognitive behavior and sensor fusion is expected to help 
such types of robots to act and behave like humans, 
have personality and take right decisions in various 
situations. 


User profile o ( 


* The robot should be able to build a user profile for each new Е 
user. 


* The robot should be equipped with the Al required to 
continuously update user profiles. 


* NLP and other tools can be used to enable the robot to 
automatically and continuously extract knowledge from users 
and update their user profiles. 


* Knowledge such as personal information, profession, city, 
country, hobbies, games they like to play, types of books they 
prefer to read, likes and dislikes, etc. 


* Oncethe robot build user profile for many users, it can start 
building family and friends networks of users and use 
information from profiles to check social connections between 
users 


Robot personality ө @ 


* Do you like your robots subservient and complimentary, Е 
or glib and a little bit cheeky? 

* you could potentially always choose a specific 
personality type for your robot that represents the kind of 
person you enjoy interacting with. 

* Robot personality could be achieved by building a user 
profile for the robot itself. 

e The robot should be able to switch between states 
according to user's mood 

* The robot should be humble and kind with kind users 
and aggressive with aggressive users 


Conclusions & discussions ө @ 


* Robots have unlimited ability to extract knowledge from 
users and unlimited storage capacity which can help it to 
track academic progress of students, social skills of 
patients with dementia and autism. 

* These data can be analysed and used to assess 
teaching methods and treatment and therapeutic 
approaches. 
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Example 1: Multi-Scheme Smartphone Positioning in known Areas 


Basic Methods Mult-Scheme Localization Multi-Scheme 
Controller 
9 Wi-Fi Fingerprinting 
9 Bluetooth Low Energy (BLE) exclusive postion | | scat sec e 
@ Pedestrian Dead Reckoning operation 
(PDR) (Sensor-based) 
@ Global Positioning System rn @ E ED | = 
ө Nearfield Communiation Schemes. “Sheme, ‘\Gehemes \schemey: "Scheme 
Hybrid Scheme: 
Absolute positioning fused with PDR 
Challenge: 
Basic 
@ Automatic Method Selection Localization | Wi-Fi BLE PBR GPS МЕС 
Methods 


@ Low power consumption 


More Information: 
Q Multi-Scheme architecture details in [Jaeger.2015] 
@ Fusion method details in [Becker.2015] 
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Coarse Positioning: Determination of Building and Floor Level 


First Solution: 
Coarse Localization Subsystem (CLS) 


Area model allows explicit mapping from 
observations to coarse location: 


@ Port Objects: Elevators, Entrance/Exit 
Doors, Staircases 


@ Mapping from radio station IDs to 
locations (WiFi, BLE) 


@ Mapping from height measurements to 
floor levels 


Drawbacks 


@ Complexity of area model 


@ Complexity of coarse localization 
algorithm 


(5) 
CDDs of adjacent 
positioning contexts 
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Using Machine Learning for Coarse Positioning 


Some Expectations 
@ CLS can "learn" localization in unknown sub-areas 
@ Adaptiveness to changes in environment (WiFi/BLE infrastructure) 


Preliminary Experiences 
@ Successful Indoor/Outdoor Classification with semi-supervised learning 


Co-Training with two Naive Bayes classifiers 

Features: GPS-Noise, light intensity, athmosperic pressure, ... 

Results: >90% correct classifications 

Easy implementation using WEKA 

Concern: Why Naive Bayes classifiers? In theory, they should not perform well! 


vvvvy 


@ Mapping observations in signal space to locations 


> Successful in small areas: Radio Maps 
> Not transferable to larger areas: 
Observation consists of pairs (sender ID, signal level) 
How to treat signals from unknown radio stations? 
» Machine-learning solutions have been proposed: complicated 


Conclusion: Partial success, no 100% reliability, non-trivial problems, dispensable. 
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Example 2: Self-Driving Cars 


Characteristics 

@ Systems rely on: Advanced Pattern Recognition, Deep Learning, ... 

9 Al technologies are crucial, indispendable. 

@ For several tasks machines are superior to human beings (e.g. Traffic sign recognition) 
Concerns 

@ What about the 100% reliability? My life depends on the system! 


@ Engineers: /t works fine, but we wonder how. 


We have to trust in machine decisions we don’t understand because of their 
complexity. This is causing discomfort. 
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Example 3: Car Brake System 


Characteristics 
@ Several functions: ABS, ESP, etc. 
"good old" traditional engineering, Al-free 
100% reliability 
C-Code, in part generated from mathematical models 


thoroughly tested 


certified 
My perception: 
@ Trustworthy 
@ 100% reliability is important: My life depends on the system! 


Concerns? 
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Example 3: Car Brake System 


Some more insights 
@ More than 600.000 lines of code 
@ More than 600 developers currently 
@ More than ... detected inconsistencies 
ө. 
Concerns 
@ What about the 100% reliability? My life depends on the system! 


ө Engineers: It works fine, but we wonder how. 


We have to trust in conventional software we don't understand because of their 
complexity. This is causing discomfort. 
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Solution? 


@ Consult a psychologist to mitigate the discomfort? 
@ Can we do without further technical progress? 
9 Is further progress possible without more complexity? 


@ Wait some more years until we can hand over all those problems to a 
"deep-minded" machine! 
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[Becker.2015] Nils Becker, Michael Jäger, and Sebastian Süß. Indoor smartphone 
localization with auto-adaptive dead reckoning. In Proceedings of the 2015 
Tenth International Conference on Systems, pages 125-131, 2015. 

[Jaeger.2015] Michael Jäger, Sebastian Süß, and Nils Becker. Multi-scheme 
smartphone localization with auto-adaptive dead reckoning. International 
Journal on Advances in Systems and Measurements, 8(3 / 4):255-267, 
2015. 
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Introduction 


IT is often presented as a solution to reduce inefficiencies in health care 


However, there is substantial evidence on MT 


" Unsuccessful implementation projects 
" |Т implementation challenges 
= Slow diffusion 


= Unforeseen CONSEQUENCES 


Context-sensitive systems 


“Context is any information that can be used to 
characterize the situation of an entity. An entity 
is a person, place, or object that is considered 
relevant for the interaction between a user and 
an application, including the user and 
applications themselves.” 


к/а, 


Context-sensitive systems 


* This definition shows the importance of which information is 
relevant or not in a context-sensitive system. 


* A context-sensitive system could, therefore, be defined as a 


system allowing interactions between multiple entities using 
relevant information 


к/а, 


Context-sensitive systems 


e “A system is context-aware if it uses context to provide relevant 
information and/or services to the user, where relevancy 
depends on the user's task”. 


* This definition shows that a context-sensitive system can 
change its behaviour and send some relevant information 
according to the context 


* A context -aware system can also learn from the user 
interaction with the system to automatically improve his/her 
experience. 

* In this manner, a context-aware system is able to provide process 


support by analysing process related data from two categories: (1) 
what is done; (2) how it is done. 


EEQ 


Context-awareness allows health IT to 
provide process support by manging the 
complexity inherent to clinical processes 
while supplying the technology with the 

process standards required to ensure 
usability 
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Thanks for listening 
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The role of adaptive mobile dedicated 


applications in the learning process of users 
with Autism Spectrum Disorders (ASD) 
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"m 
The role of adaptive mobile dedicated applications in the learning process of users with ASD 


Neurodevelopmental disorder (American Psychiatric Association, 
2013): 


* persistent impairments in social communication and social 
interaction; 


* restricted, repetitive patterns of behaviour, 
Interests, or activities; 


* impairments in social communication: Language delays 


The role of adaptive mobile dedicated applications in the learning process of users with ASD 


Technologies as support tools for persons with ASD: 


Virtual reality applications 
Robots 

Telehealth systems 
Dedicated applications 


Reference: Technologies as Support Tools for Persons with Autistic Spectrum Disorder: A Systematic Review. 
N. Aresti-Bartolome and В. Garcia-Zapirain, nt. J. Environ. Res. Public Health 2014, 11, 7767-7802 


.. 4 » 
The role of adaptive mobile dedicated applications in the learning process of users with ASD 


Technological tools designed to be used on computers, tablets or mobile 
telephones for: 


(1) Communication 
(2) Social learning and imitation skills 
(3) Other associated conditions (related to play, immagination ...) 


.. 4 » 
The role of adaptive mobile dedicated applications іп the learning process of users with ASD 


Users with ASD present good visual abilities => 


Tools based on Augmentative and Alternative Communication (AAC) 
techniques, i.e., powerful methods that combine different visual components 
in order to create syntactically and semantically correct sentences 


"m 
The role of adaptive mobile dedicated applications in the learning process of users with ASD 


There are hundreds of dedicated applications for communication, but .... 


1. Which one should we pick? 

2. Is it FREE? 

3. Is it really ADAPTIVE? 

4. Tf it is adaptive, at what cost (time, effort, technological knowledge, etc.)? 


fà 


The role of adaptive mobile dedicated applications in the learning process of users with ASD 


Note that if they are not adaptive low functioning non verbal users with ASD 
will not be able to use them!! 


Why? They need 


- "real known images" (symbolic pictures may not work) 
- "known voices" (synthesizers produce "unknown sounds") 


just to TEST the tool => too much time/effort 


Specific goals: 
- can we improve the quality of synthesizers? 
- can we produce highly adaptive free mobile dedicated applications? 


More in general: 
- Should we build totally auto-adaptive applications or should we ask, to 
a certain extent, for human intervention? 
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Human-Machine Interaction: 
lets focus on Humans for a while... 
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SEE THE USERS ME 
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THE REAL USERS 


How to investigate Monete: 


> Cattell's16 Personality 
Factors Test 


» Millon Clinical Multiaxial 
Inventory 


» Myer-Briggs Type Indicator 

» Big Five 

» Minnesota Multiphasic 
Personality Inventory 


1. Are you outwardly or inwardly focused? If you: 


• Could be described as 
talkative, outgoing 


+ Like to be in a fast-paced 
environment 


* Tend to work out ideas with 
others, think out loud 

• Enjoy being the center of 
attention 


then you prefer 


E 


+ Focus on the reality of how 
things are 


+ Pay attention to concrete 
facts and details 


+ Prefer ideas that have 
practical applications 


* Like to describe things in a 
specific, literal way 


then you prefer 


S 


Sensing 


ow do you prefer to take in information? If you: 


What's Your Personality Type? 


Use the questions on the outside of the chart to determine the four letters of your Myers-Briggs 
For each pair of letters, choose the side that seems most natural to you, even if you don't agree with ¢ 


• Could be described as 
reserved, private 


+ Prefer a slower pace with 
time for contemplation 


e Тепа to think things 
through inside your head 


* Would rather observe than 
be the center of attention 


ISTP 


Action-oriented, logical, 
analytical, spontaneous, 
reserved, independent. 
Enjoy adventure, skilled 
at understanding how 
mechanical things work. 


then you prefer 


ESTP 


Outgoing, realistic, 
action-oriented, curious, 
versatile, spontaneous. 
Pragmatic problem 
solvers and skillful 
negotiators. 


* Imagine the possibilities of 
how things could be 


+ Notice the big picture, see 
how everything connects 


+ Enjoy ideas and concepts 
for their own sake 


+ Like to describe things in a 
figurative, poetic way 


then you prefer 
y р Ef icient, outgoing, 
an lytical, systematic, 
c pendable, realistic. 


Intuition | ato run ће show and 


get things done in an 


orderly fashion. 


Playful, enthusiastic, 
friendly, spontaneous, 
tactful, flexible. Have 
strong common sense, 
enjoy helping people in 
tangible ways. 


Friendly, outgoing, 
reliable, conscientious, 
organized, practical. Seek 
to be helpful and please 
others, enjoy being 
active and productive. 


Enthusiastic, creative, 
spontaneous, optimistic, 
supportive, playful. Value 

inspiration, enjoy 
starting new projects, 
see potential in others. 


Caring, enthusiastic, 
idealistic, organized, 
diplomatic, responsible. 
Skilled communicators 
who value connection 
with people. 


INTP 


Intellectual, logical, 
precise, reserved, 
flexible, imaginative. 
Original thinkers who 
enjoy speculation and 


creative problem solving. 


ENTP 


Inventive, enthusiastic, 
strategic, enterprising, 


inquisitive, versatile. 
Enjoy new ideas and 
challenges, value 
inspiration. 


Strategic, logical, 
efficient, outgoing, 


ambitious, independent. 
Effective organizers of 
people and long-range 
planners. 


description. 


+ Make decisions in an 
impersonal way, using 
logical reasoning 


* Value justice, fairness 

+ Enjoy finding the flaws in 
an argument 

+ Could be described as 
reasonable, level-headed 


then you prefer 


T 


Thinking 


4. How do убре 


* Prefer to have matters 
settled 


«Think rules and deadlines 
should be respected 


+ Prefer to have detailed, 
step-by-step instructions 


+ Make plans, want to know 
what you're getting into 


then you prefer 


J 


Judging 


3. How do you prefer to make decisions? If yor 


+ Base your decisions o 
personal values and how 
your actions affect others 

* Value harmony, forgiveness 

• Like to please others and 
point out the best in people 

+ Could be described as warm, 
empathetic 


then you prefer 


F 


Feeling 


open 


+See rules and deadlines as 
flexible 


• Like to improvise and make 
things up as you go 


• Are spontaneous, enjoy 
surprises and new situations 


then you prefer 


P 


Perceiving 


The Questionnaire and the J-P Plan 


» The Framework 


» The J-P Plan is a reductive 
model of the MBTI 


The Questionnaire and the J-P Plan 


The Framework 


The J-P Plan is a reductive 
model of the МВТ 


The Questionnaire and the J-P Plan 


The Framework 


The J-P Plan is a reductive 
model of the МВТ 


We don't use the test for 
classifying people but 
| 


The Method A 


L 


People with different 
personalities 


e Compile the same 
Questionnaire 


Perceived 
Personality of 
Objects 


КИ oy with specific 
Users’ Personalities 


Use Case: The Design Installation 


Perceiving 


JP and the Gulfs 


» J and the Execution Gulf 
>P and the Evaluation Со 


Ambitions 


» Help designers in improving User Experience of Interactive 
Objects 

> Test this method for targeting or expanding a market 

> Auto-adjustment behavior / Personality modeling Objects 

> What would happen ii we endow a Humanoid Robot with 
such a ‘psycholigical’ capasbilitiy®... 
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Users Need to Learn the Automation 


Panel: Learning- and Knowledge-based Adaptive Human-Machine 
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Guri Verne 
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Technology influences the users' tasks 


What does automation do? 


How does it change the task(s) that are left for 
the user? 
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Guri Verne, Panel ACHI 2016, Users Need to Learn the Automation 


UiO è Department of Informatics 
University of Oslo 


Before automation 


Unclear borders for the tasks. 


Based on: Bratteteig and Verne (2016), Verne (2015) 
Inspired by: Bainbridge (1983) «Ironies of Automation» 
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Guri Verne, Panel ACHI 2016, Users Need to Learn the Automation 
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University of Oslo 


Automation introduces new tasks 


New 
task 
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Some of a different kind 


New 
task 


New task of a 
different kind. 
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University of Oslo 


Old and new tasks 
PER) 


Obsolete 
task 


Old task 


Incoherence and fragmentation seen 7 


from the User. Guri Verne, Panel ACHI 2016, Users Need to Learn the Automation 


UiO $ Department of Informatics 
University of Oslo 


To differentiate between old and new, 
users must understand the automation 


different 
automations 


To know which tasks that a user need to do, he or she will 
need to know how the automation works to be able to handle 
the tasks left with autonomy and competence. 
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Applied to Adaptive Human-Machine Interactions 


Challenges when the system/interface adapts: 


* Users /learners need to know the automation 
to understand what is happening? 


* Users need to understand the automation to 
use the system in a knowledgeable way? 


* Design for transparency 
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Trying to make an electronic lab-book: 


Observation of chemical reaction and 
visualized image of molecular world 


Akira Кио 


Department of Chemistry, 
Tokyo Gakugei University 


OH- + CH}CI — CH,OH + Сї 


Reaction Profile 
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— 
Observable Level <— > Symbolic Level <— Molecular Level 
(Macro Level < Symbols & Equations <— Micro Level ) 


Potential energy/kJmol-! 


In(k / min!) 


Phenomena Chemical concepts Molecular world 


Figure. Dividing the image into the three thinking levels 


Figure was modified from A. Ikuo, Y. Yoshinaga, and Н. Ogawa, in C.-C. Liu, et al. (Eds.), Proceedings of the 22nd International Conference on 


Computers in Education. Japan: Asia-Pacific Society for Computers in Education, pp. 489-493, 2014. 
References therein: 


[1] J. К. Gilbert and D. Е. Treagust, in J. К. Gilbert, (Ed.), MODELS AND MODELING IN SCIENCE EDUCATION Multiple Representations in Chemical 
Education, Springer, pp. 333-350, 2009. 


[2] К. Tasker, and К. Dalton in J. К. Gilbert, M. Reiner, and M. Nakhleh, (Eds.), MODELS AND MODELING IN SCIENCE EDUCATION Visualization: Theory 
and Practice in Science Education, Springer, 103-131, 2010. 


OH- + CH,CI — CH,OH + Сї 


Ехрегїепсе 
Observation 
"Procedure 
«Supplement 


Integration of Exp. & CG "Reaction Mechanism 
"Energy Change 


* Model Change 
Electronic Lab-book 


Requirements 
* Water-proof Cover 
*Small enough 


"Individual study 
"Interest> Discovery 


Figure. Developing Experimental Program and Electronic Lab-book 


Figure was modified from A. Ikuo, Y. Yoshinaga, and Н. Ogawa, in S. White, Н. Mannaert, and J. L. Mauri, (Eds.), Proceedings of 
the eighth International Conference on Mobile, Hybrid, and On-line Learning, pp. 26-27, 2016. 


NEXT STEP? 


From DigitalWorld 2016: 


“A Training-assistance System using Mobile Augmented 
Reality for Outdoor-facility Inspection” 


Yoshiki Yumbe, Osamu Segawa, Makoto Yamakita 
ACHI 6 


“Augmented Reality as a Tutorial Tool for Construction 
Tasks” 


Ana Regina M. Cuperschmid, Marina G. Grachet, Marcio 
M. Fabricio 
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